Digital correction of optical system aberrations

ABSTRACT

A digital method for removing optical aberrations from the image is disclosed. The method includes the initial profiling of an optical system and using the obtained information to correct the optical aberrations introduces to the image by the same or identical optical system.

FIELD OF INVENTION

The present invention relates, generally, to the field of opticalsystems, and specifically to improvement of the optical imaging devices.

More precisely, the invention enables correcting aberrations of theoptical system by a computational component. The invention extends theboundaries of optical design and/or range of applicability of opticalsystems by improving their quality under physical design constraints.The technology can be applied to various electro-optical imagingsystems.

BACKGROUND OF THE INVENTION

The modern imaging technology employs highly sophisticated opticalsystems, often consisting of dozens individual optical elements. Overthe past decades, imaging optics have become increasingly complex inorder to provide the light efficiency for such optical systems (e.g.single-lens reflex (SLR) cameras.) Such complexity is typically requiredfor effective usage of the available aperture ratio, as well as forcompensation for undesirable artifacts that inherent to a simple lenselement.

The elimination or minimizing of non-linear deviations (i.e.aberrations) from the ideal “thin lens” model is an important part ofthe optimization of the overall imaging system efficiency. The opticalaberrations include such artifacts as geometric distortions, chromaticaberration (wavelength-dependent focal plane), spherical aberration(optical axis distance dependent focal length), and coma (angulardependence on focus).

Since each single optical element with spherical surface(s) suffers fromthe aforementioned artifacts, the combinations of different lenselements have been used, especially when a high-quality imaging isrequired (e.g. in photography).

The complex aberration compensated optical systems that possess muchbetter geometric imaging properties, however, suffer from otherdrawbacks, including drastically increased manufacturing cost, weight,lens flare, reduced reliability, etc.

While it is possible to correct the appearance of the chromaticaberration effects using a digital post-processing in somecircumstances, in the most of the real-world circumstances, chromaticaberration results in a permanent loss of some image detail.

The proposed invention uses an alternative approach to achieve thehigh-quality imaging. Namely, instead of more complex optics, theperformance improvement is achieved by adding a pre-process orpost-process computational component to correct aberrations of theoptical system. Such optical component will be hereinafter referred toas a “digital lens” element of the optical system.

There are various methods of computational aberration correction thathave been developed and reported. Thus, the lens-profile-based imagecorrection methods typically take the known characteristics of opticallens/system into account for (automatic) correction of various types oflens distortion, such as color fringes at high contrast edges,vignetting, etc.

Indeed, the detailed knowledge of the optical system used to produce theimage could play an important role in correcting of the undesirableartifacts in the image. Due to the complexity of the chromaticaberration (relationship to focal length, etc.), the cameramanufacturers employ various lens-specific techniques to minimizechromatic aberration appearance.

Nowadays, almost every major camera manufacturer enables some form ofchromatic aberration correction, both in-camera and via theirproprietary software. Third party software tools (e.g. PTLens, DxOOptics Pro, Adobe Photoshop Lightroom) are also capable of performingcomplex chromatic aberration appearance reduction with correspondingdatabases of cameras/lens.

For example, there is a method reported in U.S. Pat. No. 6,862,373 byEnomoto, describing acquisition of both the input data from an imagetaken by an optical element (lens) and the information about the verylens been used to record this image. The method further describes imageprocessing using information about the focal length and an aperture(i.e. lens iris opening) at the time of recording, as well as lenscharacteristics to correct aberrations and vignetting in the image.

Another example, as disclosed in Japanese Patent No. 11-161773 by Habu,also describes correcting magnification chromatic aberration withoutusing any optical components. The magnification chromatic aberrationdata for the lens for each color is pre-stored, and image processingperforms enlarging and reducing the image based on the mentionedpre-stored data, thus performing the magnification aberration correctionevery time an image is obtained through this lens. Then, aftermagnification correction, the images of each color are combined into asingle image, accomplishing the magnification chromatic aberrationcorrection.

There is another method disclosed in U.S. Pat. No. 7,425,988 by Okada(and, similarly, in U.S. Pat. No. 8,508,655 by Suto) that describesmagnification or reduction of a picture on each color; a data memoryunit to store the chromatic aberration data specific to the imaging lensfor each color (including plurality of zoom, focus and aperture values)and a processing unit that controls the conversion factor and thecoordinates magnification aberration correction, using both chromaticaberration data (stored in data memory unit) and the detected image(along with the current zoom, focus and aperture values).

In an ideal situation, the post-processing to remove or correct lateralchromatic aberration would require scaling the fringed color channels,or subtracting some of a scaled versions of the fringed channels, sothat all channels spatially overlap each other correctly in the finalimage (e.g. in holographic microscopy).

In practical applications, however, even a theoretically perfectpost-processing-based chromatic aberration reduction system does notincrease the image detail in comparison to well-corrected physical lens.

From the chromatic aberration perspective, the reasons for this arefollowing: i) A computational rescaling is only applicable to lateral(not longitudinal) chromatic aberrations. ii) The individual rescalingof color channels results in some resolution loss. iii) Chromaticaberration occurs across the light spectrum, yet most camera sensorsonly capture a few discrete (e.g. RGB) color channels.

Some chromatic aberration cross-channel color contamination isunavoidable in camera sensors.

Since the above problems are closely related to the content of theparticular captured image, no reasonable amount of programming andknowledge of the capturing equipment (e.g., camera and lens data) canovercome such limitations completely.

The disclosed method proposes a new, improved non-blind deconvolutionapproach for electronic optical aberrations correction. Like the otheraforementioned methods, the disclosed method is also based on knowledge(i.e. profiling) of the optical system used for imaging. Furthermore,the method consequently processing the arbitrary captured scene with‘digital lens’ element of the present disclosure using the profile thatis already known for the imaging system.

Compared to other aberration correction techniques, however, thedisclosed profiling approach is inherently different, essentially, byutilizing a point-spread function (PSF) extraction for different imagescales (i.e. image details) and subsequent artificial neural-network(NN) training. The PSF is an important property in predicting of a lightpropagation and imaging system performance.

The disclosed method and ‘digital lens’ element expand the applicabilityof the digital lens from typical image capture systems (digital cameras)towards a broader imaging applications, including augmented reality(AR)/virtual reality (VR) display systems, headsets, viewfinders, etc.

Further features and aspects of the present invention will becomeapparent from the following description of preferred and optionalembodiments with reference to the attached drawings.

SUMMARY OF THE INVENTION

An optical imaging method and ‘digital lens’ element for removingoptical aberrations from the image is disclosed. The method is based ontwo independent procedures, with the first procedure to be performed inadvance of the second one.

The first procedure includes collecting the optical system information,performing a geometric and a radiometric alignment betweencomputer-generated charts and their image(s), extracting a PSF componentwith a far-field kernel, and convoluting it with the aligned image toproduce a far-field corrected image data. The first procedure alsoincludes preconditioning of the image data to be used for NN training,extracting NN weighting coefficients and creating an optical systemprofile, which is a data containing the NN weighting coefficients andthe far-field kernel.

The second procedure includes capturing an original image by the opticalsystem, extracting the components with low- and high-spatial-frequencies(LF and HF), preconditioning HF components for NN, also using theoptical system profile obtained in the first procedure. Thereconditioned NN output is further summed with the original LF componentforming a single corrected image with the optical aberrations removed.

The method can be used with various optical systems, such as cameras,viewfinder eyepieces and displays with known manufacturer parameters.The method can be applied on segments (tiles) of the images.

The LF and HF components can be calculated using linear low-pass andhigh-pass filters, respectively, while the preconditioning can include adynamic range reduction and application of nonlinearity. The NN outputcan include multiple pixel values while NN training can be performedusing a Levenberg-Marquardt algorithm.

The inverse pre-conditioning can be applied to reduce NN training error.The NN output can be calculated using an arithmetic inverse of the HFpart, while PSF component extraction can be performed usingtwo-dimensional Fast-Fourier transform.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1: General view of the optical aberrations correction method.

FIG. 2: Geometric and radiometric alignment of charts and images.

FIG. 3: Optical system profiling.

FIG. 4: Far-field image correction and point spread function (PSF)extraction.

FIG. 5: Image processing

FIG. 6: Example of using a Bayer filter with a pixel quad (RG/GB) forthe NN output data.

FIG. 7: Optical aberrations correction for display systems.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The disclosed invention is generally serves the task of correctingoptical aberrations using the a non-blind deconvolution technique thatis described further in detail. The term ‘non-blind’ is used in a sensethat the invention utilizes a point spread function (PSF) which is known(e.g. pre-estimated in advance) by “profiling” the optical system ofconcern. In general, the PSF describes the (3-dimensional) response ofan imaging system to a point-like object or point light source. Infunctional terms, PSF is a transfer function of the imaging system in aspatial domain.

The general layout of the disclosed invention is shown in FIG. 1, wherethe scene (1) is observed by the optical system (e.g. camera) (2), whichcaptures the digital images (3) of the scene (1) and provides thecorrected image (5) of the scene (1) after applying a digital aberrationcorrection post-processing, performed by a computational component (4)(i.e. ‘digital lens’) to correct aberrations of the optical system (2),in assumption, that the computational component (4) is provided with thespecific information (6) (i.e. ‘optical profile’) about the opticalsystem (2) in advance. Such specific information (6) is obtained by theprofiling the optical system (2) and plays a critical role in thedisclosed invention, as will be explained further in detail.

The disclosed method is based on separating the full procedure ofcomputational correction of optical system aberrations into two separate(independent) steps, as follows:

Procedure I:

Profiling of the optical system of choice. In this part the informationof how to optimally correct the aberrations induced by this opticalsystem is obtained. This information is then stored in a form of the“optical profile” for this particular optical system. Typically, suchprofiling of the optical system is a time-consuming, iterative process.Fortunately, it only needs to be performed once for the optical systemof choice (e.g. once it is designed and manufactured).

Procedure II:

Correcting the aberrations of the optical system by using the “opticalprofile” obtained during the Procedure I. This step is non-iterative,relatively fast, enabling to perform the optical aberrations correctionof an image, for example, at video signal framerates, either on existingimage processors and/or in a power-constrained mobile devices.

Such separation of the procedure into two steps also allows extendingthe method's applicability range, for example, from image capturesystems (digital cameras) to more general optical imaging applications,including various modern AR/VR displays/headsets systems, viewfindereyepieces, etc.

In the preferred embodiment of the invention, the Procedure I can becarried out in optical system of choice by replacing the position ofhuman eye with a high-quality image-capturing device. Consequently, anyfuture image that produced by this optical system can be correctedroutinely in the Procedure II, essentially, canceling the aberrations(intrinsic to such known, imperfect optical system) out.

Thus, the preferred embodiment of the invention includes two, generallyseparate procedures: Procedure I,—profiling of an optical system,specifically, obtaining the ‘optical profile’ containing the informationabout the optical system, and Procedure II,—applying the optical profileobtained in the Procedure I to an image being captured by the same (oridentical) optical system in order to reduce the optical aberrations inthe captured image.

During the Procedure II, the so-called “digital lens” correction isapplied directly to the data (image) obtained by the optical system(i.e. camera) in Procedure I. Corrected image can passed through theimage signal processing in a the same way as one without the “digitallens” correction.

The Procedure I of the preferred embodiment of the invention divided, inturn, into 2 major steps, as described here in detail with reference toFIG. 2.

Step 1: Profiling Images Capture and Geometric/Radiometric Alignment.

In this step, described in FIG. 2, a set of computer-generated charts iscreated. These charts are either printed on paper or displayed on ahigh-quality/high-resolution display, to be captured through the opticalsystem/camera.

Then, the computer-generated charts are captured by an optical system tobe profiled (also shown as (3) in FIG. 1), producing a sequence of,generally N, images (21).

The captured sequence of images (21) includes the knowledge of theoptical system's parameters (i.e. metadata of the camera used during thecapture), such as focus distance, aperture size (lens iris value), focallength, etc.

Then, the geometric and radiometric alignments (23) of the charts (21)from the sequence of captured images is performed. For this purpose, asubset of profiling images from the sequence (21) and their capturedversions (output of (22)) is used to facilitate the geometric alignment(i.e. scene/features alignment) and radiometric alignment (i.e.estimating the nonlinear function of color values in each image andapplying it to align images radiometrically, so that the color valuesare consistent for all images of a sequence).

Note, that during such alignment procedures, the computer generatedcharts (21) are aligned with the captured images, i.e. the versions ofthe chart images captured by the system being profiled. All suchcaptured images are captured with the same camera settings and undersame illumination conditions, so they do not require alignment to eachother.

There are various established image processing techniques for geometricand radiometric alignment have been reported in the prior art which canbe utilized by the disclosed method with the goal of providing anaccurate (as possible) image alignment in terms of matched colors andgeometric features of the charts.

The output of this alignment process (23) is a pair of aligned images(24). In a preferable embodiment if the invention, a subset of computergenerated charts, geometrically and radiometrically aligned to theircaptured versions is provided as a part of an output (24) of thealignment process. As shown in the FIG. 2, such pair includes so-called‘Aligned’ and the corresponding ‘Captured’ image.

These images (24) (i.e. aligned charts and their captured versionstogether) are further used in creation of the optical system profile,specifically, for extracting knowledge about the way by which theaberrations of the particular optical system will be corrected.

Step 2: Optical Profile Creation for the Optical System.

This Step 2 of the Procedure I is explained in in details below and usesthe ‘Aligned’ image and the ‘Captured’ image from the previous step toperform an actual system profiling, see (24) in FIG. 2.

A complete process of profile creation (i.e. Procedure I) for an opticalsystem is depicted in FIG. 3. The process starts with a pair of inputimages—aligned and captured (31), which are (optionally) divided intotiles (32) (described later in detail in Step 2a with the reference tothe following FIG. 4). This data is passed to a far-field correction(34), which includes the far-field image correction and point spreadfunction (PSF) extraction (described later in detail in Step 2b with thereference to the following FIG. 4).

The far-field correction (34), in turn, provides two outputs: far-fieldkernel coefficients stored into optical system profile (33) and pixeldata that has only near-field aberrations remaining.

These pixel data follows through the LF/HF separation (36). The outputsof (36) are provided in form of the inputs for the NN (39). For thispurpose, the pixel data containing LF spatial frequencies is passeddirectly, while the pixel data containing HF spatial frequenciesundergoes a data conditioning (38) before being transferred to NN (39).

The coordinates of the pixel data (either within the image plane orwithin the tile), which is provided directly from (32) (or (31) iftiling step is not used) forms another input of NN (39).

During the Procedure 1 (profiling) the NN (39) requires a target pixeldata to be trained. Such pixel data is provided from the aligned imagethrough HF filter (35) and data conditioning (37). The NN weights andbiases (391) obtained by the training are also stored in the opticalsystem profile (33).

The process of far-field image correction and point spread function(PSF) extraction is shown in the FIG. 3 as the procedure (34), anddescribed later in more detail, in connection to FIG. 4.

Step 2c. With the reference to the FIG. 3, the data of Aligned andCaptured images (31) (or their aligned tiles (32)) is separated into twoparts, according to the data spatial frequencies, as follows:

One part (35) will only include the content with thehigh-spatial-frequencies (hereinafter the HF part extracted from thedata), while another part (36) will have both the HF part and an LF part(the content with the low-spatial-frequencies, extracted from the data).This separation can be realized by application of well-known linearfilters that based on a low-pass and high-pass kernels, respectively.

Importantly, the LF part (36) is not typically affected by aberrationsand, therefore, can be passed as-is, without processing. The HF part(35), however, needs to be properly corrected in order to obtain adesirable improved image.

As mentioned, the LF part (36) and the HF (35) part are both extractedseparately for the ‘Aligned’ and the ‘Captured’ images, or theirrespective tiles, if the option of dividing the images into the tiles ischosen.

Step 2d. With the reference to the FIG. 3, the previously obtained pixeldata from the HF part is prepared (i.e. ‘conditioned’) by theconditioning procedure (37) to play a role of an input data (i.e.reference data) for the NN training (39). Typically, the NN (39) is usedin a machine-learning procedure to estimate (or approximate) functionscoefficients (391) that depend on a large number of, generally, unknown,inputs. In image-processing, the NN can use the reference (input) imagefor self-pretraining procedure, which isknown.

Various known data preconditioning techniques can routinely be usedbefore the launch of the NN training. Reconditioning (37) (datamanipulation after the NN training) can also be used to improve the NNtraining efficiency, e.g. to reduce a NN training error.

Reconditioning (37) is similar to inverse of pre-conditioning steps(38). For example, when the dynamic range of NN data is reduced in someway, the output of the NN should be reconditioned with correspondentdynamic range expansion.

In the preferred embodiment of the invention, such data (preconditioningby (38) and post conditioning by (37)) include any combination of thefollowing two procedures:

-   -   1) A reduction of the dynamic range of the data. Generally, NN        operates optimally when it is provided with the data having a        limited dynamic range. There are many well-known ways to reduce        the dynamic range of the data. One possible way is to normalize        the input data by either a sum of input pixels' values or by a        maximum input pixel value.    -   2) Application of nonlinearity to the pixel's value. Such        nonlinearity serves to emphasize the contribution of pixel        having small values, hence instructing the NN to produce a more        accurate result for small-amplitude inputs. This, in turn,        results in a better signal-to-noise ratio (SNR) in the output        image. For example, the introduced nonlinearity can have a form        of Â(1-alpha), where: A—is a pixel value and alpha—is a small        constant (typically in the range of 0.02-0.2).

Once trained, the NN algorithm (including data pre- andpost-conditioning) will obtain the information (33), i.e. “OpticalSystem profile” about how to convert the aberrated (e.g. blurred) imageinto the one that is as close to the perfect (without aberrations) aspossible.

The NN (39) operates in a pixel-wise fashion. Accordingly, in order tocorrect aberrations for a particular pixel, a certain neighborhood(within the input captured image (32)) of this pixel is processed. Inthe preferred embodiment of the invention, the data is prepared for theNN inputs and outputs, as described in details with reference to FIG. 3.

The NN (39) is provided with the knowledge of the processed pixel'slocation, effectively allowing the NN (39) adapting to variability ofthe PSF over the given tile. For this purpose, the pixel coordinates(31) are passed to the inputs of NN (33) from the tiling step (32) takenfrom the Captured image (34).

The HF part (36) from the Captured image tiles (32) is passed to theinputs of NN (39) after being subjected to aforementioned ‘dataconditioning’ (38). This HF part (36) is extracted using pixels within acertain neighborhood (e.g. less than nine pixels distance from the pixelbeing processed) of the captured image.

The HF part (35) of pixels being processed from the aligned image tiles(32) is also passed to the outputs of NN (39) after being subjected toaforementioned ‘data conditioning’ (37) and serves as an NN output data.

Moreover, the LF part (36) from the Captured image tiles (32) is passedto the inputs of NN (39). As mentioned, this LF part (36) is extractedusing pixels within a larger neighborhood (e.g. farther than eightpixels distance from the pixel being processed) of the captured image.

It should be noted that in some embodiments of the invention, the pixelinput data is obtained straightforwardly from a mosaic-filtered sensor,without any pre-processing that provided by imaging system signalprocessor (ISP). For example, the Bayer filter can be used, which is acolor filter mosaic array applied to photo-sensor matrix to arrange RGBcolor filters in a square grid pattern. Such arrangement of colorfilters is used in most single-chip digital image sensors found indigital color cameras, camcorders, and scanners. In such case, it ispossible to use such type of NN that produce multiple pixel values inoutput at once (quad pixels in case of the bayer pattern), as will bedescribed later in connection to FIG. 6.

Once all the input and the output data is prepared, the NN training islaunched.

In the preferred embodiment of the invention, the training is performedusing well-known training algorithms, such as Levenberg-Marquardt, forexample. Weights of the trained NN (marked (391) in FIG. 4) areextracted and stored in the form of the ‘optical system profile’ alongwith the kernel obtained from the far-field correction step (34). Theweights and kernel are unique per each tile, so each tile has it's ownset of such parameters.

As mentioned, the preferred embodiment of the invention includes two,generally separate, procedures: the Procedure I, which produce the‘optical system profile’ and the Procedure II—which applies the profileobtained in the Procedure I to an image being captured by the same (i.e.identical) optical system, in order to reduce the optical aberrations ofthe captured image.

The complete process of profile creation (the Procedure I) for anoptical system was explained with the reference to FIG. 3. The far-fieldimage correction (34) and point spread function (PSF) extraction (Steps2a-2b) are described here in more detail, with the reference to FIG. 4.

In the Step 2a, the images (provided by field of view of the camera,initially shown as (24) in FIG. 2) can optionally be divided intosmaller parts, hereinafter called the ‘tiles’ of the Captured image (42)and the Aligned image (41), respectively, using a Tile Separationprocedure (40).

The aforementioned PSF is highly variable by its nature due to itsdependence on a size, concentration and distribution of various featureswithin the optical system field of view. Accordingly, the (optional)steps (41) and (42) are generally performed to reduce such variabilityof PSF shape within the field of view of the given optical system.Within each single tile, the PSF variability (i.e. aberrations) is lower(although still present) and, therefore, can be more effectivelycompensated, compare to the original (Captured and/or Aligned) images.

In the Step 2b, so-called remote correction is performed, which isexplained below in connection to the FIG. 4. The PSF approximation isextracted (43) and then separated into two parts: the near-field part(44) calculated within a certain neighborhood of a processed pixel (e.g.with radius of less than six pixels); and the far-field part (46), wherethe PSF (and aberrations) is extracted outside of the near-part radius.

The mentioned extraction of the PSF approximation (43) from the inputdata can be based on one of the many methods known in prior art, see forexample Felix et. al. While this reported method provides a veryaccurate PSF estimation, it is very time consuming. For the sake ofreducing computational load, a simpler method would be much preferred.

In the preferred embodiment of the disclosed invention, the followingapproach is implemented for (43): i) A two-dimensional (2D) Fouriertransform of the tile of the Captured image is divided by a 2D Fouriertransform of the corresponding tile of Aligned image. ii) The result istransformed back with 2D inverse Fourier transform.

Such approach is well-known to one experienced in the art. It worthmentioning that in order to avoid the potential noise over-amplification(i.e. low-amplitude image frequency of the Aligned image), a guardingbias value(s) should be added to the values of the frequency componentsof the Aligned image.

The far-field kernel calculation (45) and convolution (47) result in animage with the aberrations and blur caused by the far-field part arecorrected. The parameters of the chosen far-field correction (i.e. thekernel coefficients (45) obtained from the far-field correction (46))are recorded for a later use, along with the optical system profile, asdescribed below.

In image processing, a kernel is a convolution matrix (much smaller thanimage itself) that is used for modification of the image (e.g.sharpening, blurring, edge detection, etc.). This is accomplished byapplying a matrix convolution operation between a specific kernel and animage of choice.

Near-field part of the PSF (44) is discarded, because a simple reversionfor the near-field PSF is not possible without a significant noisemagnification of the output image.

The reason for aforementioned separation (44) and (46) into thenear-field and the far-field is twofold. First, the far-field correctiondoes not amplify the noise, eliminating undesirable artifacts, and,therefore, can be carried out by relatively simpler methods that requireless computational resources. Secondly, such near-field correctionrequires a limited amount of input data to be processed, hence furtherreducing the processing requirements. Such separation is performedarithmetically. In a simplest way, all PSF values within certain radiusare forming the near-part, while all PSF values outside of this radiusare forming the far-part. In practice though, there is some smoothtransitional region between these two parts.

To have such a smooth transition, a frequency domain approach ispreferable and a two-dimensional fast-Fourier transform (2D FFT) of thePSF is calculated.

The far-field part (46) is obtained by multiplying frequency componentsby the window which has a zero amplitude at high frequencies whileapproaches a unit value at low frequencies. A Gaussian 2D distributionwould be a convenient example of such window.

A 2D inverse transform is employed to obtain the far-field part (36) ina spatial domain.

Since the far-field corrections do not have any high-frequency spatialcomponents, its application does not amplify any high-frequency noisecomponents. Therefore, a far-field kernel construction procedure (45)and can be straightforwardly realized by some well-known methods.

The method used in the preferred embodiment utilizes a convolution ofthe input image with the obtained far-part PSF, following by subtractionof the result from the input image. Such (relatively simple) approachwould effectively correct the first-order blur and aberrations caused bythe optical system.

It should be noted that so-called secondary aberrations can still beinduced due to the fact that the captured image (used for processing) isalready aberrated, However, since in typical optical systems, thefar-field effects are low in amplitude, the secondary effects of suchaforementioned correction are negligible.

When the far-field kernel (45) is constructed, its coefficients arepassed to the optical system profile. Then, the kernel is convoluted(47) with the captured image tile pixel data, resulting in anotheroutput of the far-field correction (shown as (34)-(36) connection in theprevious FIG. 3). This output is the tile that contains only near-fieldremaining aberrations (48).

The Procedure II is the application of the described ‘digital lens’ tothe captured image, as explained here in detail with reference to FIG. 5for the preferred embodiment of the invention.

The original captured image (52) and optical system profile (56) are theinputs for the Procedure II processing. Original image's pixel data is(optionally) separated into tiles (54). It further undergoes thefar-field correction (55) using the kernel coefficients from opticalsystem profile (56). Afterwards, the pixel data goes through LF/HFseparation (57) with its outputs being the inputs for NN (590).

Thus, the pixel data containing only the LF spatial frequencies ispassed directly, while the pixel data containing HF spatial frequenciesundergoes the data conditioning (591) before being transferred to NN(590).

Yet another input to NN (590) is the coordinate of the pixel data(either within the image plane or within the tile), which is provided bythe (54) (or directly by the (52) if the tiling procedure is not used).

Weights and biases of NN (590) are obtained for a given tile from theoptical system profile (56). Output of NN (590) undergoes the datare-conditioning (593) and is summed up (58) with the LF output obtainedfrom LF/HF separation filter (57). The summed result is furtherrecombined (53) merging the individual tiles into a single image (on iftiling step was used). Such merging forms the corrected output image(51).

The processing of an arbitrary captured scene (while the profile (56)was already been prepared for this system in the Procedure I (FIG. 4,(43)) is performed using processing similar to the Procedure I (opticalsystem profiling), with only a few changes in a data-flow.

The input (52) is the captured scene to be processed is passed throughthe tiling procedure (54) in a similar manner to one described in theProcedure I, Step 2a, see FIG. 3 and FIG. 4.

Then, the far-field correction (55) is performed in a similar manner toone described in the Procedure I, Step 2b, see FIG. 4, with thedifference that the far-field correction kernel is already known asbeing obtained in the Procedure I, Step 2c.

Then, the image data is separated into the LF part and the HF part (57),similarly to the LF/HF separation procedure explained in the ProcedureI, Step 2c, see FIG. 3.

The HF part is further passed through the pixel data′ conditioningprocedure (591), similarly to the conditioning explained in theProcedure I, Step 2d, see FIG. 4.

The described procedures (52), (54), and (55), along with the opticalsystem profile (56) obtained in the Procedure I, form an input data forthe NN (590). This data is similar to one obtained in the Procedure Iduring the system profiling step in terms of pixel coordinates. Asbefore, it also has the LF, and HF part of pixels processed within thedifferent neighborhoods.

The following processing of the output of NN (590), ultimately, formsthe image (51) (through the tile merging (53)) which is a desired image,in a sense that the optical aberrations have been removed. Specifically,the NN (590) output value is undergoing the data reconditioningprocedure (592), which, in the preferable embodiment of the invention,realized by ab arithmetic inverse of the data conditioning procedure(590). This result is subsequently summed by the (58) with the LF part(57) of processed pixels.

Finally, all processed tiles (53) of the input image are re-combined, toform a single output image (51). In the preferred embodiment of theinvention, the tiles combined in (53) have some overlapping areas. Inthese areas, the result of processing (53) is smoothly attenuated fromone tile to another, to eliminate any visible seams during their merginginto the final image (51). Such visible seams are caused by atile-to-tile deviation from the corrected output.

In another embodiment of the invention, speed optimizations of thedigital lens processing can be performed. Thus, with the reference tothe FIG. 6, in order to output a single pixel, the pixel's neighborhood(61) in the input image is processed by applying the NN (62). The size(e.g. a ‘diameter’ (61) chosen at the profiling procedure) of thisneighborhood depends on optical system characteristics. For example, ifthe Bayer filter is used for the input and output data, then thefour-pixel quad (RG/GB) (63) can be computed at once.

In yet another embodiment of the invention, the monitor (either display,virtual reality headset or a viewfinder eyepiece) can be used for theprofiling procedure of the optical system. In such embodiment, themonitor is used to display profiling images; thehigh-quality/high-resolution camera is placed precisely at the locationwhere the observer eye will be during normal operation; and theprofiling images are captured with this camera. Then, the optical systemprofile is created, similarly to the method described above.

With the reference to the FIG. 7, in order to obtain a corrected imageduring normal operation of such embodiment, the image (71) is initiallypassed through the digital lens correction (72), and then transferred tothe display device (73). The corrected image is then observed (74) bythe observer.

Further improvements of the VR headset include eye pupil positiontracking (by an eye-tracker) for the optimal performance. Completeoptical system in this case includes the eye pupil with, generally,arbitrary (non-optimal) location. Knowing the position of the eyerelatively to the rest of the optical system allows an accuratecorrection of aberrations that arise in such system.

Although several exemplary embodiments have been herein shown anddescribed, those of skill in the art will recognize that manymodifications and variations are possible without departing from thespirit and scope of the invention, and it is intended to measure theinvention only by the appended claims.

What claimed is:
 1. An optical imaging method for removing opticalaberrations, wherein a captured image being captured by an opticalsystem, comprising: a first procedure and a second procedure, whereinthe first procedure is performed in advance of the second procedure, thefirst procedure is comprising of: collecting an optical systeminformation, extracting a point-spread function (PSF), separating thePSF into parts, the parts being at least a near-field part and afar-field part, producing a far-field correction, training aneural-network (NN), extracting a weighting coefficients of the NN, andcreating an optical system profile; and the second procedure iscomprising of: creating an original image in the optical system,applying the far-field correction, calculating a NN output, and forminga corrected image, wherein the corrected image is the original imagewith reduced optical aberrations.
 2. The method of claim 1, wherein thefirst procedure further comprising: collecting the optical systeminformation, wherein the optical system information is a focus distance,an aperture size, and a focal length, creating a set ofcomputer-generated charts, creating a sequence of captured images of thecomputer-generated charts by the optical system, performing a geometricalignment and a radiometric alignment between the sequence of capturedimages and the computer-generated charts, producing an aligned imagefrom the geometric alignment and the radiometric alignment, and creatinga close neighborhood and a remote neighborhood within the aligned image.3. The method of claim 2, wherein the near-field part being calculatedusing the close neighborhood, and the far-field part being calculatedusing the remote neighborhood.
 4. The method of claim 2, wherein thefar-field correction being a convolution of the far-field kernel withthe aligned image, the far-field corrected image containing onlynear-field aberrations.
 5. The method of claim 2, wherein the opticalsystem information includes lens manufacturer parameters.
 6. The methodof claim 2, wherein the first procedure further comprising: separatingthe aligned image into an aligned low-spatial-frequency (LF) partcontaining low-spatial-frequencies being calculated using the closeneighborhood, and an aligned high-spatial-frequency (HF) part containingonly high-spatial-frequencies being calculated using the remoteneighborhood, calculating a preconditioned aligned HF part using thealigned HF part, and providing a first input data for NN, wherein thefirst input data is the combination of the preconditioned aligned HFpart and a captured LF part.
 7. The method of claim 6, wherein thealigned LF part is calculated using linear low-pass filter and thealigned HF part is calculated using linear high-pass filter.
 8. Themethod of claim 6, wherein the preconditioned aligned HF comprises acombination of dynamic range reduction of the aligned LF part andapplication of nonlinearity to the aligned LF part.
 9. The method ofclaim 2, further comprising extraction of captured tiles from thecaptured image, and extraction of aligned tiles from the aligned image.10. The method of claim 9, wherein the forming of the single correctedimage includes merging corrected tiles.
 11. The method of claim 10,wherein the merging of the corrected tiles includes elimination of seamsbetween the corrected tiles.
 12. The method of claim 1, whereinfar-field kernel coefficients calculated from the far-field part. 13.The method of claim 1, wherein the optical system profile being a datacontaining the NN weighting coefficients and the far-field kernelcoefficients, wherein the optical system profile being unique per eachaligned image.
 14. The method of claim 1, wherein the second procedurefurther comprising: creating a close neighborhood and a remoteneighborhood within the original image, separating the original imageinto an original LF part containing only low-spatial-frequencies beingcalculated using the close neighborhood, and an original HF partcontaining only high-spatial-frequencies being calculated using theremote neighborhood, calculating a preconditioned original HF part usingthe original HF part, providing a second input data for the NN, whereinthe second input data is a combination of the optical system profile,the preconditioned original HF Part and the original LF part,calculating a reconditioned NN output from an output value of the NN,and performing a summation of the output value of the reconditioned NNwith the original LF part.
 15. The method of claim 14, wherein thereconditioned NN output is calculated using an arithmetic inverse of apreconditioned captured HF part.
 16. The method of claim 1, wherein theoptical system is a camera.
 17. The method of claim 1, wherein theoptical system is a viewfinder eyepiece.
 18. The method of claim 1,wherein the optical system is a display.
 19. The method of claim 1,wherein the output value of the NN has multiple pixel values.
 20. Themethod of claim 1, wherein a reconditioning is used to reduce an errorin the step of training of the NN, wherein the reconditioning beinginverse to a pre-conditioning.
 21. The method of claim 1, wherein theseparating of the PSF is performed using 2D FFT.
 22. An optical systemfor removing optical aberrations, wherein the optical system has anoptical profile, the optical system comprising: a lens which forms acaptured image having optical aberrations, an array of photosensitiveelements disposed within the optical system, a conversion component thatconverts the captured image into a digital image, and a computationalcomponent configured to form a corrected image by removing the opticalaberrations from the digital image through corrections based on theoptical profile, wherein the optical aberrations comprising longitudinalaberrations.
 23. A headset having a display, wherein the headsetcomprising: an optical system having an optical profile, wherein animage produced by the optical system has optical aberrations, and acomputational component configured to form a corrected image by removingthe optical aberrations through corrections based on the opticalprofile, wherein the optical aberrations comprising longitudinalaberrations, and the corrected image is displayed on the display. 24.The optical system of claim 22, wherein the optical profile is formed bycollecting the optical system information, extracting a point-spreadfunction (PSF), separating the PSF into parts, the parts being at leasta near-field part and a far-field part, producing a far-fieldcorrection, training a neural-network (NN), extracting a weightingcoefficients of the NN, and creating an optical system profile; andwherein the corrections are applied by creating a digital image,applying the far-field correction, calculating a NN output, and formingthe corrected image.
 25. The headset of claim 23, wherein the opticalprofile is formed by collecting the optical system information,extracting a point-spread function (PSF), separating the PSF into parts,the parts being at least a near-field part and a far-field part,producing a far-field correction, training a neural-network (NN),extracting a weighting coefficients of the NN, and creating an opticalsystem profile; and wherein the corrections are applied by creating adigital image, applying the far-field correction, calculating a NNoutput, and forming the corrected image.